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ABSTRACT 

The relationship between embedded support devices 
(ESDs) and study results was examined to determine whether ESD£ ha^e 
a positive effect on study outcome* Two versions of a learning unit 
of a governmental law course at the Open University of the 
Netherlands were constructed. One represented the original text and 
the other was about 40X shorter as a result of deleting most support 
devices (schemes, summaries, and learning objectives). Thirty-six 
students from the University of Maastricht studied one of the course 
versions. Tests and questionnaires were developed to measure effects 
of the two versions and the potential interrelation with independent 
variables. No effect of studying different course versions was found 
on study time, stuc'ent evaluations of the learning access level, and 
"knowledge" questions. But students who studied the course version 
with ESDs had significantly higher scores on insight questions. Of 
the independent variables (e.g., prior knowledge, time used, reading 
comprehension, educational level, and gender) only reading 
comprehension had a significant effect. Three tables and two figures 
prei,ent study data. Two appendixes contain an additional figure and 
table of supporting information. (Contains 29 references.) 
(Author/SLD) 
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Varying embedded support devices in a course: what is the effect? 



Abstract 



In this investigation the relation between embedded support devices (esd) and study results is 
investigated. The key question is: do ESD have a positive effect on study outcome? Two versions of 
a learning unit (chapter) of a course of the Open university of the Netherlands named 
'Introduction to govenunental laW were constructed. One was an almost exact copy of the original 
course text The second version was approximately 40% shorter as a result of deleting most 
support devices (such as schemes, summaries and learning objectives). This was done without 
omitting basic information or affecting the coherence of the text Thirty-six students from the 
University of Maastricht (RL) studied one of both course versions. Several tests and questionnaires 
were developed to measwe the effects of course versions on the study results and the potential 
interrelation with independent variables. 

No effect of studying different course versions was found on study time, student evaluations of 
the learning access level and 'knowledge' questions. But students who studied the course version 
with ESD obtained significantly higher scores on 'insighf questions. Interrelations with the 
independent variables: prior knowledge, time used, reading comprehension, educational level and 
gender were tested. These variables were added to the ANOVA models as covariables. Only 
reading comprehension turned out to have a significant effect. 
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1 Introduction 



In a context of distance education the quality of study materials is of prime importance. The study 
materials of the Open university of the Netherlands (Ou) are still mainly text-based. Much energy 
and time is invested to generate these materials. An essential part of the printed study text consists 
of embedded support devices (e.g. content pages, examples, advanced organizers, pre- and 
postquestions, schemes, tests, margin texts). Embedded support devices (esd) are defined as formal 
and /or content-related add-ons, extensions and elaborations of the learning content in printed 
study materials. They aim at scaffolding the learning process by supporting orientation, planning, 
processing and evaluative aspects of the individual learning process. 

The project 'Course Versions' (Cursusvarianten) of the Centre for Educational Technology and 
Innovation (cnc) focuses on the effects of course versions on study outcomes. Especially, the effect 
of varying the amount of ESD is of main interest. The key question oip this project is: *ls the high 
investment in the design and elaboration of printed study materials worth the effort?" Where 
earlier research focused on researching the actual use of ESD, the key questions of the 'Course 
Versions' project are related to the 'effects' of esd on study outcomes/results. 

The research method used in our earlier research was based on questionnaires and interviews (see 
section 2 for more information). An important disadvantage of these two methods.however is that 
they den^nd introspection and reflection on cognitive functioning. A great deal of the study 
process takes place at a subconscious level (Resnick, 1987). Using interviews or printed 
questionnaires therefore incorporates the risk of underestimating or mistaking functions and effects 
of ESD. Just & Carpenter (1977, p.l) state: "For most adults, comprehension is rapid, automatic and 
effortless. But, despite of its apparent simplicity, comprehension includes a myriad of 
subprocesses, each of which by itself constitutes a formidable computational task. (...) All these 
processes occur so quickly that their richness is underestimated. Because of the sheer speed of 
comprehension, it is difficult for ti\e reader to explain how he came to understand a passage. That 
task is left to those who study comprehension." 

It is difficult to get access to study processes by means of an interview or questionnaire. To ' 
overcome this problem other research methods are needed. Two alternatives are the eye-movement 
registration method or measuring product outcomes, e.g. study results. The eye-movement 
registration method is in this perspective very efficient since the attentional focus of students on 
specific text parts is registered, without the student being conscious about this. Focusing on 
product outcomes can consist of measuring what and how many students recall and comprehend 
after studying a text. We used in the actual research both different methods. The research based 
on the eye-movement registration method will be reported separately. The research in this report 
focuses on the effect of ESD on study results and text comprehension* To use this research method 
we developed two versions of a study text: a study text with embedded support devices and one 
without embedded support devices. 

This research report continues with chapter 2 where an overview of earlier research is presented. 
Chapter 3 includes a summary of the general theoretical base and background in order to present 
in a coherent framework the research hypotheses. Chapter 4 focuses on a description of the 
research design. Attention is paid to the careful selection of the research population, the 
development of the course versions, the research instruments, the research procedure and the 
processing of the data set for further analysis. An analysis of the research data based on ANOVA 
is presented in chapter 5* In chapter 6 we discuss the results and present some conclusions. 
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2 Earlier research 



This section focuses first of all on earlier research conducted outside ttte context of the Ou of the 
Netherlands. It thereby presents some general results of this research body and especially indicates 
some important shortcomings of this research. In a second subpart an overview of research 
conducted at the Ou of the Netherlands is presented. 



2.1 Research conducted outside the Ou of the Netherlands 
Limted amount of research 

Clyde and Crowther (1983, p.4) dearly point at this shortcoming when they state: "Despite the 
growth of distance education, little systematic knowledge is available concerning the ways in 
which students use the distance teadiing materials provided to them." (cf. also Marland, Patching, 
Putt & Putt, 1990). Despite the high investment in ESD by course developers, little is known about 
there functions and effects in study materials. This is particularly true if we focus on research in a 
'distance education' context. 

Doubts about functions and effects of esd 

Most ESD are designed to evoke some kind of behaviour of students. Wade and Trathen (1989, p. 
40) however state: "(...) despite their popularity, there is no consistent empirical evidence that 
these techniques are any more effective than more passive methods of reading-only or repetitive 
reading...". Even when - from a theoretical point of view - functions and effects are assigned to 
specific ESD, there is the problem that Meyer and Watson (1991) have signalled: practical 
implications derived from this are often not useful. 

Use of BSD 

Many research on ESD is marred by the fact that researchers hardly know what learners do when 
they are presented with text-embedded support. It is for instance not sure whether or not they use 
or know how to use these support facilities (cf. Winne, 1983). We may not forget that the student - 
when working at a distance with printed learning materials - has complete veto power over 
leaminfr (Rothkopf, 1970, p.326). Winne (1983, p245) states for example when he tries to explain 
the inconsistencies in earlier research results : "these results may reflect different ways that learners 
cognitively responded to the instructional stimuli rattier than weaknesses in their effects". In order 
to obtain more valid results, this researcher trains his students to use the embedded support 
devices (objectives, adjunct postquestions). A similar approach is adopted by Bernard (1990) (cf. 
de Jong & Simons 1988, p.l78). 

The question is not only whether students use the support devices or whether they know how to 
use them, but also whether they need ttie support provided. They are only useful if students need 
them. Waller (1979, p.l80) mentions e.g. different user objectives when dealing witii study texts: 
entertainment, recapitulation /reviewing, browsing, studying in deptti, searching for a particular 
item, etcetera. Each user perspective affects the potential effects/functions of BSD. 

Methodological problems 

Valcke & Martens (1992, p.8-9) stated that the ecological validity of many research set-ups can be 
questioned. Research is hardly set up in a distance education context or at least in a natural 
context. Most research is set up in isolated lab-like settings. 

Theoretical problems 

Finally, serious theoretical problems can be put forward when analyzing the research in the 
literature. Valcke & Martens (1992) signalled a lad<: of theoretical base, reductionist approaches and 
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behaviouristic influences. They indicated also the fact that potentially important variables are 
excluded from theoretical models, conceptions and approaches. 



22 Research at the Open university of the Netherlands 

At the Dutch Ou, research in relation to ESD has been set up since 1991. We shortly describe some 
of the research-approaches and a subset of the results. 

Theoretical framework 

A basic phase in the Ou-research was the elaboration of a theory on the potential functions and 
effects of embedded support devices in printed study materials (Valcke & Martens, 1992). This 
theoretical model is considered as a starting point from where to look at the actual use of ESD by 
students when they study the course materials. 

In reviewing the literature, also methods and instruments were examined to investigate the use, 
functions and effects of embedded support devices. 

Cheddisi 

A checklist was developed to map the occurrence and quality of ESD in a study text, at course and 
at learning unit level (Valcke, Daal, Martens & Dochy, 1991). 

This checklist was used in a try-out with the course Introduction to Governmental Law' and 
proved to be useful for systematic mapping ESD (Martens & Daal, 1992; Martens, Daal & Valcke, 
1991). 

Research with interviTtv method 

In a first research set-up 25 law students of the Ou were interviewed extensively about theiv study 
and the use and effects of ESD (Poelmans, Martens, Daal, Valcke & Dochy, 1992; Valcke, Martens, 
Poelmans & Daal, 1993). The interview method had the advantage to reveal considerable and 
detailed information. However interviewing students, tanscribing, analyzing and categorizing the 
student answers was very time-consuming. This implied to restricting the number of interviewees: 
25 law students. Conclusion of this research indicated that ESD are used and highly appreciated by 
students. Furthermore students who use ESD received significantly higher final test scores and 
needed less attempts to take the final test. 

Research with the questionnaire method 

At the Ou of the Netherlands an annual evaluative study of the department Education and 
Evaluation (Onderwijs en Evaluatie or O&E) involving about 2000 students is set up and is always 
based on questionnaires. Students taking courses in different subject domains participate in the 
study. In the year 1992 a part of this cjuestioimaire consisted of questions about embedded support 
devices, their use and links with study success. In subsequent research this part of the 
questionnaire was used. With this instrument compared to *e interviewmethod a greater amount 
of students could be contacted. Researchers of O&E and OTIC analyzed the student answers 
(Martens, Poelmans, Daal, van Staa & van Meurs, 1993). This research revealed that ESD are 
extensively used and highly appreciated. Only students of the subject domain Cultural Sciences 
appreciated ESD to a lesser extent. No interrelation with the use of ESD and study success could be 
found. Study success was operationalised as the speed to attain study credits. 
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Theoretical background 



This section summarizes the theoretical base of this research. First of all key processes and 
variables are presented. These are in section 3.3 integrated into one consistent theoretical model. 
We refer to Valcke & Martens (1992) for a more extensive description and discussion. 



3.1 Cognitive psychological framework 

Much effort was invested in developing a theoretical framework about embedded support devices 
in relation to studying from printed texts. An essential part of this theoretical framework is based 
on considerations about the nature of cognitive functioning. We opted for instance for the 
componential theory of Sternberg (1985, 1988) to describe and explain cognitive functioning in our 
specific distance education context. Sternberg's cognitive processing approach stresses, in one 
consistent framework, the importance of a large complex of cognitive processes. The theory can be 
linked to intra-individual variables and particulariHes of the instructional setting of the Open 
university: distance education, self-instruction, independent learning, etcetera. 

Cognitive functioning is - according to Sternberg - described with the concept 'components'. A 
component is "an elementary information process that operates upon internal representations of 
objects or symbols" (1986, p225). Sternberg distinguishes three groups of components: 

- Performance components (PC); 

- Knowledge acquisitJori components (KAC); 

- Metacognitive components (MC). 

Metacognitive components control, evaluate and plan the activities of fl\e other components. They 
are executive processes. The knowledge acquisition components are involvvKl when new 
information is integrated in already available information in memory. This integration process 
implies encoding, combining and comparing. Performance components are titose processes that 
operate when a task is actually executed. Knowledge acquisition and performance components are 
non-executive processes. The metacognitive components can be perceived as "the stage directors 
that tell the actors - the performance and knowledge acquisition components - how to act The 
actors in turn provide feedback to the directors about how the show is going. (Sternberg, 1988, 
p.l89)". 



3.2 Reading: an intervening process 

In our conception of independent learning at a distance, major importance is paid to the in-built 
support in the printed study materials. We prefer to describe the learning process- as a recursive 
process in which cognitive operations continuously interact in a network. 
In our view, ESD influence specific (sets oO cognitive processes and can therefore result in a 
positive effect of exhort a specific function. 

The interference of the reading process in our research cannot be neglected and is even to be 
incorporated into our model. We thereby foais on those aspects of the reading process that are 
related to studying, learning. As a consequence we did especially focus on the comprehension of 
printed texts (discourse comprehension) and not on the decoding aspects of the reading process, 
(discourse). 
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33 



Other variables 



Other variables of importance are: 

non-cognitive aptitudes such as motivation 

task environment the task environment refers to those variables that define the setting in 
which the student studies. These variables show a great variability in a distance education 
setting. 

individual variables: individual variables s^ich as prior knowledge also play an important 
role in the study process. 

Figure 1 summarizes the theoretical framework. 



MC 



KAC * FC 



tMdfa)( pOMM 




tcSc BvllQBUJlBBlt 



tsil-^BSD 



Figure 1. A model for learning from printed study materials 



3.4 



Towards a research model 



The model in Figure 1 is useful to derive hypotheses for specific research set-ups. In the study 
reported here, we investigate the effect of ESD on study outcomes. We constructed two course 
versions: one with the 'normal'^ set of ESD and one with as few as possible. 

Study outcomes as a result of studying one of the two versions were measured in 3 different ways: 
time used for studying, 

comprehension of the studied learning unit. The comprehension score will be split in two 
components: a score for insight and a score for knowledge, 
Student rating o^ ^.2aming Access Level (LAL) 

Considering the theoretical framework, intervening variables can play a major role. Despite the 
complexity of these variables a specific set of such variables are included: 
reading comprehension 

individual variables: Previous research indicated that gender, prior knowledge and 
educational level are important in relation to studying in distance education (Poelmans et 
al, 1992; Valcke et al, 1993; Dochy, 1992). Time used for studying is not only considered as 



*Nonnal* refers to the amount of BSD normally included in the original learning unit of the couise. 



ERLC 



12 



BhST COPY AVAILABLE 



Vaiying embedded support devices in a course: what is the effect? 



a result of the text version used (dependant variable) but also as a student characteristic: 
some people tend to study (read) longer than others. In the latter case the tim^. used to 
study the course text can be treated as an independent variable. We have approached this 
variable in both ways. 

Note that in the research model (Figure 2) no distinction is made between the cornponents of the 
learning process (MQ KAC, PC). In this investigation we regard the learning process as a whole. 

Linking the former to our theoretical model, the following more operational 'research model' can 
be presented: 




oQgDittvo tsptitodaa 



piDOOBS 



ICadfalg ptOOQBI 



3 



indMdaftl 



* stu^^ tiOlO 



TV 



tBxtivi&BSD 



test ivil^ ESS 



Figure 2: Research model 

Translating this research model to practice implies the following. As described in section 22 in 
eariier research (Poelmans et al, 1992; Valcke et al, 1993) we found that the use of ESD improves 
study success. It also was found that ESD are highly appreciated. However ESD take a lot of place 
(about 40%) of the total content of Ou course materials (see section 4.1 for a detailed description), 
so it can be expected that studying a text with less ESD takes less time. This reasoning is false if 
the supporting functions of ESD can decrease study time. In that case no differences in study time 
between the two versions are expected. 

Considering this, the following hypotheses will be tested. These hypotheses are formulated as 
scientific hypotheses that are to be rejected 

Research hypotheses 

1. Studying from text without ESD (short version) leads to more comprehension of the 
learning unit (knowledge and insight). 

2. Studying from text with ESD (long version) leads to less time spent on studying. 

3. Studying from text with ESD (long version) leads to lower appreciation from students of 
the learning access level (LAL). 

The impact of the following 5 intervening variables will be taken into account: 
reading comprehension 
gender 

prior knowledge 
educational level 
study time 

8 
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4 Research design 



In this research design we decided to focus on product outcomes: what and how many students 
recall and comprehend after studying a text, and is there a difference in time used for studying 
and in student ratings of learning access level. To set up this research design we (1) developed tv-o 
versions of a study text: a study text with and without embedded support devices, (2) defined a 
research sample of students, (3) developed research instruments and (4) composed a research 
procedure. We end this section with subparts focusing on the quality of the research mstrument? 
used and the research data obtained. 



4.1 Two versions of a learning unit of the course Introduction to Governmental Law' 

Two versions of one learning unit of the course 'Introduction to governmental Uw' were 
developed: an 'extended' and a 'shorf version. 

The 'extended' version resembles the learning unit of the course as pubhshed by the Ou. Only a 
few adjustments were added to make it possible that students can study t^s specific le^ng umt 
apart from the other learning units in the course. Prior analysis (Martens & Daal 1992; Martei^ et 
al 1991) reveals that the learning unit selected for our research can be considered as representative 
fo'r the learning units in this particular course. A learning unit is estimated to take about 4 hours of 

study time for a novice learner. , , , . ^ ^«,„voe 

The 'shorf version contains almost no embedded support devices. All embedded support devices 
that can be easily eliminated witiiout affecting the coherence of tiie learning unit intent are 
removed. Embedded support devices that are of importance for tiie coherence of the study text or 
that contain basic mformation are not removed. As a result only the ESD are removed tiia can be 
disconnected from the basic text. The amount of removed ESD consists of almost 6 of the 15 
oriKinal pages (40%). Table 1 contains an overview of tiie type and number of the embedded 
suS,ort devices that have been removed. In appendix A a page from both versions is mcluded to 
give the reader an impression of the important differences between the two text versions. 

Table 1: Type and number of removed embisdded support devices 



embedded support devices 


number 


content page 


1 


learning objectives 


1 


italics 


12 


margin text 


24 


scheme 


3 


example 


11 


summary 


1 


feedback 


4 


question 


4 



The text reductions as a result of removing the ESD do not decrease the amount of 'govermnental 
1'. In our opinion, the omission of tiie ESD affect stiidy-process featiires such as: 



law information' 
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orientation, planning, processing and testing. But, also information about links of the law content 
to 'real life' is cancelled by removing the examples. 



42 Research sample 

This research involves students taking courses in the law domain of the Ou of the Netherlands. 
But, students in a distance eduction context are difficult to reach for research purposes (due to 
travel time and the combination of work and study tasks). For this research we therefore selected a 
comparable sample of students from a population studying at a non-distance university, the 
university of Maastricht (RL). Despite the differences in study context, these students are 
experienced with self study because of the educational system they are involved in (problem 
oriented learning). Moreover, before the start of the experiment, the research sample was given a 
short introduction about functions and effects of ESD and the organization of Ou course material. 
By means of advertisement first year law students were asked to participate. A letter was sent to 
these students who wished to take part, confirming the date and place for the research. Thirty-six 
students enter this research. This sample consisted of 16 male and 20 female students. The mean 
age of the students was 20 years. All students had prior knowledge of the study field 
'(Governmental law' at an introductional level. . 



4.3 Research instruments 

Considering the amount of variables in our research model, a variety of research instruments was 
developed to gather specific information/data: 

to obtain a measure for tiie co-variable 'reading comprehension' we constructed a reading 

comprehension test; 

prior knowledge of the law domain was tested with a prior knowledge state test; 
the co-variables gender and educational level were extracted from part A in the 
questionnaire; 

study outcomes were measured with a comprehension test 'Governmental Law' 
the determination of Learning Access Level (LAL) was based on part B of the 
questionnaire; 

the variable 'time' was measured by recording the time subjects took for studying the 
course. 

More details on each specific instrument are given in the following section. 



4.3.1 Reading comprehension test 

In the section on the theoretical base it was already indicated that reading comprehension is an 
important component of our model for learning from printed study material. To measure reading 
comprehension differei\t possibilities are available: 

self-ratings: The student is asked to value his own reading comprehension. Possible 
questions are 'flow many pages do you read in one hour?" or "Do you skip an article in 
the paper, because you didn't understand it?" Answering these questions is not easy. 
Students interpret these questions in different ways. There is also the possibility that 
students give socially desirable answers. 

administering an existing reading comprehension test: The problem is that no Dutch 
reading comprehension test is available at academic level. Research is currently done in 
this field by de Geus, Henneken and Rietsma (1992) but no results/instruments are 
available at this moment. 

a test in a different language: translation of a test developed in a different language. 
Translation is difficult and affects the reliability of the new 'translated' test. 

10 
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developing a ne^vv test Regarding the disadvantages of the former possibilities, developing 
a new test is tl\e only acx:eptable alternative. For this research a reading comprehension test 
at academical level was developed. The first version of the test consisted of 23 text 
fragments (e.g. texts of courses of the Open university). Each text was followed by two 
propositions about which a student had to judge the congruence with the text. A try-out of 
the test resulted in few adjustments. The new test version was probed and adjusted anew. 
This resulted in a final version consisting of 22 text fragments. This version of the test had 
a reliability rate of .69. For a detailed description of the development of this test, we refer 
to Martens (in prep.) 



4.3.2 Prior knowledge state test Law 

To measure the prior knowledge state in relation to the law domain, an adjusted version of a final 
test for the 'Basic course Law' was developed. This was done in dose collaboration with a law- 
expert. The nim\ber of test items was reduced and several items were rewritten. The final test 
consisted of 32 items: 20 multiple choice questions and 12 true/?/false questions. 



4.3.3 Questioimaires 

The questionnaire comprised three different parts: 

Part A consisted of general questions about motivation, age, study habits, educational 
level, gender etcetera. 

Part B contained questions about the course text version a student studied. Students are 
asked to judge the clarity, view and difficulty of the text version on a five points scale. These 
ratings give an indication of what we call the rating of the Learmng Access Level (LAL) of the text 
version. 

Part C included questions about embedded support devices. The embedded support 
devices of the extended version of the learning unit were reviewed one by one. These questions 
focuses on the use of the support devices, their appreciation and their functions and effects. Part C 
is only relevant for those students who studied the 'extended' version. 



4.3.4 Comprehension test Governmental Law 

The comprehension test measures the mastery of the objectives pursued by studying the specific 
learning unit. The original version of the lemming unit comprised a test, consisting of open-ended 
questions. These questions were used as a base to develop a final test for the experimental 
versions. The final test consists of 23 questions: 16 tru3/?/false questions and 7 multiple choice 
questions. Analyzing these questions from a didactical perspective two subsets of about the same 
length can be distinguished: 

'knowledge questions (13) checking the straightforward recapitulation of certain domain 

aspects 

and insight questions measuring a more thorough insight in the domain (10). 
The test covered in a balanced way all learning objectives pursued in the learning unit. 



4.4 Research procedure 

The research took place during two half day-periods. The location was an auditorium of the 

university of Maastricht. The following procedure was used: 

The first half-day period 47 students participated. The reading comprehension test, the 
prior knowledge state test and part A of the questionnaire were administered. 
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After the first research session we analyzed the data obtained. The results of the prior 
knowledge state test were used to match two groups of students to be involved in the next 
research activities. Two levels of prior knowledge were considered in this matching 
procedure: high and low prior knowledge. 

Two weeks after the first session the students were invited to participate in the second 
session. The research population consisted of 36 students^ of which 18 studied the 
'extended' version of the learning unit; the other 18 studied the 'shorf version. As stated 
earlier, these two groups were matched considering their prior knowledge level. During 
the session students studied one of the two versions. The time students spent studying tiie 
learning unit was recorded. After studying the learning unit students were asked to solve 
the comprehension test and answer part B and C of the questionnaire. To motivate the 
students an extra financial reward of 25 Dutch guilders was promised to the two students 
with the best comprehension test score. 

45 Preparation of the data-set 

To analyze the data a codebook was compiled describing how the student responses were to be 
coded in an unambiguous way. This resulted in a set of 181 research variables. The data set was 
entered into the computer to be processed with the spss^ statistical package. Scores for several 
variables were obtained by following this process: 

Scores of respectively the prior knowledge test, the reading comprehension test and the 
comprehension test Governmental Law were computed. This resulted in three variables: 
total score of prior knowledge, total score reading comprehension and total score 
comprehension Governmental Law. The comprehension test governmental Law consisted 
of Icnowledge' and 'insighf items. The sub-score for Icnowledge' questions and 'insighf 
questions was also computed. The variables which gave an indication of the use of a 
specific embedded support device were as well computed. The new variable gives an 
indication of the general use of embedded support devices. 

The variable 'student rating of Learning Access Level' (LAL) was constructed by combining 
the ratings to the different items of questionnaire B. 

A set of variables was receded to reduce the amount of response categories: 

The reading comprehension score and prior knowledge Law score were recoded into two 
groups: high versus low prior knowledge and high versus low reading comprehension. 
The variable 'general use of embedded support devices' was also recoded into two 
categories: much versus little use of embedded support devices (by means of frequency 
analysis we made sure that the categories contained about the same number of students). 
The time spent studying the learning unit was recoded into three different categories: 
(lowest thru 42 minutes = 1), (43 minutes thru 52 minutes = 2) and (53 minutes thru 
highest = 3). 

The variable 'educational level' was recoded into two different categories: students with 
higher education experience and students without higher education experience. 



4.6 Reliability and validity of the research instruments 

Next to a screening related to the ANOVA-assumptions also the determination of validity and 
reliability measures of the research instruments is needed. The following considerations are 
important in this perspective: 



From the initial sample (N=47), eleven students participated in a separate research set-up, involving eye- 
movement r 4stration. Results of ttils research are reported elsewhere 
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All items of the reading comprehension test should measure the same aptitude (internal 
consistency). Items that correlate negatively should not be part of the test. Crombach's 
alpha of the remaining items should be at least .65. 

The comprehension test and the prior knowledge test consist of two or three subsets of 
items testing different mastery levels (knowledge versus insight/apply). Internal 
consistency is not expected to be high because of the quite different subjects of the 
questions. Before calculating Cronbach's Alpha, both tests have been split into subparts. 

We discuss the results by reviewing each research instrument separately: 
Reading comprehension 

After deleting all items with negative correlation with the total score, 29 items remained. 
Crombach's Slpha was .68 (standardized). 
Learning access level (LAD 

A total score of the student rating of learning access level (LAL) was computed by adding 
all scores of part B of the questionnaire together. No item-analysis was preformed. 
Prior knowledge test 

We indicated that internal consistency is expected to be low, considering the special nature 
of this test The reliability coefficient of the knowledge questions was .14 (17 items), for the 
insight questions it was .18 (12 items) and for the apply questions (which is a special case 
of insight questions) it was -AO (3 items). No further reliability analysis was performed. 
The content validity was e^^sured by choosing/adapting an existing test in cooperation 
with an expert in the law domain. 
Comprehension test 

We repeat here the problem mentioned in relation to the prior knowledge state test. 
Internal consistency is expected to be low, considering the special nature of this test. The 
reliability coefficient of the knowledge questions was .12 (13 items), and for the insight 
questions it was .08 (10 items). Again no further reliability analysis was performed. 
The content validity was also ensured by choosing/adapting an existing test in close 
collaboration with a law expert. 



4.7 Some special variables 

Two variables in this research have to be discussed separately: time and use of the embedded 
support devices. 

Time is not only to be considered as a result of the text version used (dependent variable) but also 
as a student characteristic: soma people tend to study (read) longer than others. In the latter case 
the time used to study the course text can be treated as an independent variable. We have 
approached this variable in both ways. 

We also administered a small questionnaire about the use of ESD as perceived by the students. 
Since only students who studied the version with ESD could answer these questions we only have 
data from one from the two groups. Therefore the questionnaire only had an explorative function 
and will not be referred at in the results section. 



4.8 Test statistics 

The analysis technique used in the following section is analysis of variance. Considering ANOVA 
as the statistical technique to be used, espedally assumptions have to be met in relation to the 
dependent variables. 

To make sure that the general assumptions for executing an analysis of variance are met, the 
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following tests have been used: 

Cochran Bartlett test for homogeneity of variance of Ae dependent variables. 
Normal plots of tiie distributions of the dependent variables. 

Kolmogorov-Smimov Goodness of fit test to check the assumption that the dependent 
variables are approximately normally distributed. 

Bartlett Test of sphericity. If the dependent variables are uncorrected ANOVA instead of 

MANOVA must be used (Norusis, 1988). 
The results indicate the following. The variances are significantiiy unequal. All dependent variables 
are approximately nonnally distributed. Since tiiere are no significant correlations between tiie 
dependent variables ANOVA in stead of MANOVA will be used. This means that the inequality of 
variances is no longer a problem (For the exact figures we refer to appendix B). 

The scores for each test, related to independent variables, were screened for outliers. No problems 
were met. 
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5 Results 



5.1 Introduction 

In this section the results of the analysis of variance will be presented. We will investigate which 
hypotheses can be rejected based on the analysis of the results. The influence of 5 intervening 
variables will be checked by means of matching and/or analysis of covariance. 

5.2 Analysis of variance 

First the results of the analyses of variance are presented. Next, we focus on the influence of the 
covariables. We will examine if changes occur in the main effects as stated below. 



Table 2: Effects of version (ANOVA; n=36; oneway probability) 



variable 


average 


average 


degrees of 


F-value 




short 


long 


freedom 






version 


version 






LAL 


46,17 


46,50 


1;34 


.024 


time 


46,50 


51,56 


1;34 


.901 


insight 


6,44 


7,56 


1;34 


3.47* 


know 


8,00 


8,28 


1;34 


.167 



• P<0.05 



From Table 2 it can be concluded that 

ESD have a positive effect on the comprehension of printed study texts. Students that read 
the 'extended' version with ESD obtain significantly higher scores on insight-questions. 
This effect cannot be explained by longer reading times of the text with ESD. Although the 
text without ESD was about 40% shorter, no significant differences in average reading time 
are perceived between the two groups of students studying different text versions. 
Students that study the 'extended' version need on average 51,56 minutes and those 
studying the 'shorf version study on average 46,5 minutes. 

No significant differences are found in students ratings of the Learning Access Level 
(LAL). 



5.3 Analysis of covariance 

In the cells of Table 3 the 'new' F-values from the main effects are displayed that are assessed after 
adjusting for the covariables that are printed in the left coUum. In this analysis a new score of the 
dependent variable is predicted from an 'intewening' variable. 
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Table 3: F-values from the main effects after adjusting for cx)variables 
(d.f. 1,35; n=36; oneway probability) 



(co)variable 


LAL 


time 


insight 


know 


gender 


,062 


2.22 


2.9* 


.167 


prior knowledge 


,014 


.850 


4.64* 


.268 


time 


.004 




3.7* 


.177 


reading comprehension 


.096 


1.29 


2.61 


.099 


educational level 


.099 


.665 


3.0* 


.131 



• P<0.05 



From Table 3 can be concluded that there are almost no significant changes of the main effects: 
only when reading comprehension is used as a covariable there is no longer a significant relation 
between version and score on insight questions. Reading comprehension is therefore an 
intervening variable that influences the score on a test for comprehension. 

Both student groups in the research design were matched according to their prior knowledge. 
Therefore the results indicate that prior knowledge turns out to be an intervening variable of little 
importance. Gender, educational level and time also seem to have no significant impact. 

The results in Table 3 must be interpreted vdth caution because it is difficult to meet with all the 
assumptions underlying tiie ANCOVA model (e.g., error free measurement of the covariable). 
Also, (he changes in significance level are very small. The p-value from the main effect, when 
corrected for reading comprehension is still p = 0.058. It must also be kept in mind that the 
reliability of the reading comprehension test was low (Alpha .68). Inspection of the correlations 
between all the variables used, showed that there was only one significant correlation: female 
students tend to study longer than male students (r=.50; P<0.01). There was no significant 
correlation between reading comprehension score and the score on insight questions of the 
comprehension test (r=.24). Therefore we conclude that reading comprehension is a significant 
intervening variable in this study, despite the randomization of the groups, but that its influence is 
to small to reject the effects printed in Table 2. 
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Discussion and conclusions 



Considering the results in Table 2, conclusions can be presented in relation to the research 
hypotheses: hypothesis 1, 2 and 3 can be rejected: 

1. Studying from text without ESD ('sborf version) leads to more comprehension of the 
learning unit (knowledge and insight). 

2. Studying from text with ESD ('extended' version) leads to less time spent on studying. 

3. Studying from text with ESD ('extended' version) leads to lower appreciation from students 
of the learning access level (LAL). 

Considering the analysis of co-variance (Table 3), the independent variable 'reading 
comprehension' can be retained as an important intervening variable. 

In our opinion the results found are not surprising. They are in Une with results previously found: 
Poelmans et al. (1992) and Valcke et al. (1993) showed that studems use ESD, that they 
appreciate ESD positively and moreover that using ESD is related to better study results 
(higher scores on examinations and less attempts required to get a sufficient result). This 
research was based on interviews with 25 students. 

The results found in this study are also in line with Martens et al. (1993) and van Staa & 
van Meurs (1992) who showed also that students do use and appreciate ESD. However 
studies, based on questionnaires (N=1159), showed no interaction with study results. They 
argue that this is probably due to different operationalisation of the variables and /or a 
different research design. 

We found not as many interactions with student characteristics as Valcke et al. and Poelmans et al. 
This might be due to our research design. First, student characteristics were only measured by - 
short- questionnaires. Second, our research population was relatively homogenous. This might 
mask the impact of certain relations. The rationale for selecting this specific sample of students was 
based mostly on the fact that students in a distance education context are hardly available for this 
kind of research. . 
Nevertheless, one covariable had significant effects on the results: reading comprehension, future 
research is needed to find out to what extent reading comprehension is an import variable in 
distance eduction, aUke for instance prior knowledge (Dochy, 1992). To make more solid 
conclusions on this subject further improvement of the reading test (alpha) is required. 

In general we conclude that this investigation revealed information about the relations between the 
variables as displayed in Figure 2. These relations were in the directions we expected. 

As to the two study text versions used in the research design, some critical remarks can be made 
that also can explain some results. Both versions were based on the original Ou-materials. This 
implied that we didn't change or optimize the quality of the ESD in the text. A thorough 
investigation of the material showed that some critical remarks could be made about the way ESD 
were implemented in this course (Valcke et al., 1991; Martens et al., 1991; Martens & Daal, 1992). It 
?5 possible that improving the quality and inaeasing the quantity of ESD in the course would 
strengthen the positive effects perceived. Nevertheless, the results found support further 
investment in research and practice on ESD in shidy texts for distance education learning materials. 

This shidy focused on 'printed' shidy materials. Despite the fact that 'prinf is still a predominant 
mode of delivering the educational materials, further developments are on their way. Printed 
materials are not 'flexible', the embedding of the support devices cannot be varied easily according 
to shident needs or demands. More flexible learning materials, delivered in an elecb-onic way, 
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open prospects for mrther developments in this field. At this moment research is carried out at the 
Ou of the Netherlands, Among others Portier & Valcke (1993) have for instance developed a 
prototype of a flexible and interactive learning environment In this learning environment, 
embedded support devices can be added to or removed from the study text in a flexible way 
depending on student needs/demands. 

Finally, when adding the results of this study to these of previous studies a more clear picture 
starts to arise about the potentialities of ESD: investing time and money in the implementation of 
ESD seems worthwhile since ESD are used and seem to have a positive effect on study outcomes. 
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Appendix B 

inspection of dependent variables 



test/variable 


time 


know 


insight 


LAL 


variables 

COg6Cll6r 


Cochran Bartlett test for 
homogeneity of 
variance 


.548; 

AD/ 


.590; 


.665; 

1 ni n 
1.912 


.662; 
l.oaU 


boxs M 
19.6* 


Normal plots 


normal 


normal 


normal 


normal 




Kolmogorov -Smimov 
Goodness of fit test 
(normal) 


.557 


.992 


1.07 


1.082 




Bartlett Test of 
sphericity 










7.26 with 6 
d.f. 



» p <0.05 
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